This enables significantly faster generation.
Here is a simple test I ran.

# following above code
data = sentences * 128 # total 256 sentences
model.cuda();
data = [' '.join([x]*10) for x in data] # make the prompt longer to be more realistic
from tqdm.auto import tqdm

def test(batchsize = 1, max_gen_len = 20):
    for i in tqdm(range(0, len(data), batchsize)):
        batch = data[i: i+batchsize]
        inputs = tokenizer(batch, return_tensors="pt", padding=True)

        output_sequences = model.generate(
            input_ids=inputs['input_ids'].to(model.device),
            attention_mask=inputs['attention_mask'].to(model.device),
            do_sample=False, # disable sampling to test if batching affects output
            pad_token_id=tokenizer.eos_token_id,
            max_length=len(inputs['input_ids'][0]) + max_gen_len, # let it generate longer
        )
        outputs = [tokenizer.decode(x) for x in output_sequences]


%time test(1, 20)

%time test(32, 20)

%time test(1, 100)

%time test(32, 100)

Hey @cccntu - this is a great addition! I very much like your appraoch here.
I also checked that all GPT2 SLOW tests function correctly and added a test to make sure batch generation works as expected!

With the current implementation, the user would not be able to define his own position_ids for generate, since they are always overwritten in the prepare_input_ids_for_generation, but I think this is OK because:

1. Previously, it was impossible for the user to use position_ids because they would have to be extended by 1 each generation step - a feature which is not implemented
2. I don't see any reason why position_ids should be different from the way it is implement in the PR right now

@LysandreJik - this feature was heavily requested by the community (linked a couple of issues below) and I think this is a great way to handle GPT2 batch generation. What do you think?

Related issues: #6742, #4746,
#4824

@cccntu - Great work on this PR! If this PR is merged and you want to help the community a tiny bit more, you could give a short description (similar to what you've done above) on how to do batch generation with GPT2 here: https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517. Many people have been asking for this so they would be very glad to see a short forum post about it.

Thanks a lot again!

Awesome, great work @cccntu ! It would be amazing if you could write a little description of how your PR works on the forum: https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517 - the community would be very thankful I think :-)

@patrickvonplaten Thanks for the suggestions! I just added some description to the forum post. 😃

link to the post for future reference: https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/2

Can you please add batch inferencing for GPT2DoubleHeadsModel too?

@patrickvonplaten @cccntu

I can see how batch generation is now available. I was wondering, if there's already a way to do the same but with different arguments of max_len & min_length per encoded_text in a batch in model.generate(). Goal here is to generate new text for a batch of encoded text with variable size.

Hi @spate141,

Did you mean passing a max_len & min_length as n-element array?
It would fail here:

You can also checkout the discussions in #3021, or the forum post: https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/3

Did you mean passing a max_len & min_length as n-element array?

• Yes, exactly! Instead of single int values for all texts in a batch... an array of values for each text in a batch.

I saw the code and I can see why it will fail. #3021 seems informative, I'll take a look.

Meanwhile I found this way to get what I mentioned:

• Let's assume a model accepts input of max_len = 64 and we want to generate new text for a piece of text of size 300 tokens.
• Since we know what's the max_len is, we have make sure that we split our input text into 5 batches: [64, 60, 58, 50, 56, 12].
  • This was done in some clever way to ensure that each text segment follows valid grammar rule and also don't go above that max_len limit.
• For all these 6 text segments we want to generate new text with following min, max values:
  • min_values: [100, 100, 100, 100, 100, 25]
  • max_values: [120, 120, 120, 120, 120, 50]
• To do that, I can just pass a global min & max values (i.e. 100, 120 respectively) to model.generate() along with a tokenized batch of input text segments.
  • input_ids_shape: (6, 64), min_len: 100, max_len: 120
• My only issue here is regarding last text segment in a batch of (6, 64) tokenized tensor. Ideally, we want new generated text of size min of 25 tokens and max of 50 tokens. Generating a new text of size 100 tokens from an input of 12 tokens will be gobbledygook.
• To handle this, I can just take the last segment of generated text that belongs to our last input text; and split the text and discard everything above its ideal original min/max limit, i.e. (25, 50)

OR

• I can just go with doing same but I combine first 5 text segments and generate text on (5, 64) and generate text for the last one (1, 64) in two pass

OR

• I can just generate everything in 6 pass for each 6 text segments and pass their ideal individual min/max limits

@cccntu In your 2nd comment to this pull request, you posted some impressive results on why doing batch_generation is ideal, specially let's say when you have a GPU. I'm just trying to figure out if doing the same in my case is worth the latency when I have to do some post-processing. I'll post some latency results once I have this setup ready.

Update: @cccntu

I went with my 1st approach where I'm generating text for all texts in a single batch with global min, max values. In most cases where my last text chunk in batch is smaller meaning its min/max values are smaller than rest of text chunks in a same batch; I'm just trimming tokens. Results are impressive so far. Some numbers just in case someone stumble upon this thread in future:

Fixed size text batches:

• This shows when passing list of text chunks as single batch tensor Vs passing text chunks as individual in for loop. max_len, min_len variables are kept same in both. Y-axis shows total time in seconds for model to finish generating text.
• All the text chunks are of same size.

Variable size text batches:

• Same as above, but here I'm using variable size text chunks.
• For example: 2 Long, 1 Short means my input is 2 long size texts + 1 short size text. This is to test what happens when I'm generating text for variable size text chunks in a single batch.
• Also to note that I'm trimming generated text for short text chunks in post processing. So, time on Y-axis include that.

Overall, batch text generation seems very useful(🎉) despite one has to add some overhead on top to manage some use cases.

@cccntu Thanks for your great work! I stumbled upon this thread and would like to know:

1. Would this batching mechanism works for GPT-NEO?
2. Would this batching mechanism works for pipeline inference?
   If so, is there any changes or considerations I need to do or know?

Thanks for the code! I wonder if now I could generate sentences in a batch withother models (BertGeneration, for instance)? Looking forward to your reply!

@cccntu Thanks for your code. By using the correct position_id in this case, we can do batch inference in pytorch model now.

But when we export the gpt2 model to onnx with GPT2OnnxConfig

onnx_config = GPT2OnnxConfig(model.config)
## or using past_key_values mode
# onnx_config = GPT2OnnxConfig(model.config, use_past=True)

Then the onnx model inputs don't contation position_id but only input_ids nand attention_masks。
So we can't do correct batch_inference with onnx model now, right？

Thank you for the code. I wonder if you have tested whether there is performance drop when using batch generation? Especially when the GPT-2 model is finetuned with right-padded data.